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Components inputs and Outputs 

This section provides further details of the component 
inputs and outputs used in the system according to the 
embodiment. 

5 

Log Collector (LC) 

input from LCM: 

System_conf iguration 

Retrieval_Interval={default=24 hrs | hourly 

io interval=l - 24 hrs} 

Cleanup„Interval={ default=7 days [ weekly 

interval =1 - 7days) 

Output to LCM: 
15 Log transfer list 

LC_Name {FQHN, IP address} 
SD_Name {FQHN, IP address} 
Date 

Retrieval_Interval 

20 Time 

Files={f ilel, file2, f ile3 . . . ) 
Errors 
file 1 
file 2 
25 file 3 

Log Collector Manager (LCM) 

Input from DAM: 

System_conf iguration 
30 Retrieval_Interval={default=24 hrs | hourly 

interval=l - 24 hrs} 

Cleanup_Interval*-{ default=7 days | weekly 

interval=l - 7days) 

LC_List= {LC_Narael, LC_Name2 , LC_Name3 . . . } 
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LC_Name 'n'={FQHN, IP address} 

SD__List={SD_Namel, SD_Name 2, SD_Name 

3 . . . ) 

SD_Name ' n ' = { FQHN , IP address} 
5 LCM_status_request /* request status update of LC 

log archiving managed by LCM */ 



Input from LC: 

Log transfer list 
10 LC_Name {FQHN, IP address} 

SD_Name {FQHN, IP address} 
Date 

Retrieval_lnterval 
■ Time 

15 Files={f ilel, file2, f ile3 . 

Errors 
file 1 
file 2 
file 3 

20 

Output to SM : 

Log transfer list 

LCM_Naine {FQHN, IP address} 
LC_Name {FQHN, IP address} 
25 SDjName {FQHN, IP address} 

Date 

Retrieval_lnterval 
Time 

Files={f ilel. file2, f ile3 . 
30 file 1 

file 2 
file 3 



Output to DAM: 
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Log archival transaction complete 
LCM_Name {FQHN, IP address) 
LC_Name {FQHN, IP address} 
SDJSIame {FQHN, IP address} 
5 Errors 

LCM_archival_complete /*when all logs have been 
transferred to the SM for that interval*/ 
LCM_status_update 

LC_List= {LC_Namel , LC_Name2 , LC_Name3 . . - } 
o LC_Name • n 1 = { FQHN , IP address, 

status= [archived | cached | waiting] } 

Storage Manager (SM) 

Input from LCM: 

Log transfer list 

LCM_Name {FQHN, IP address) 
LC_Name {FQHN, IP address} 
SD_Name {FQHN, IP address) 
Date 

Retrieval_Interval 
Time 

Files={filel, file2, f ile3 . . . ) 
file 1 
file 2 
file 3 

Input from DAM: 

System_c on figuration 

Archival_Duration={ typel, type2 , type 3 . . . } 
type'n'={online= [number_months] , 
of f line= [number_months] } 
Log_Location_Request 
SD_Type 
SD_Name {FQHN} 



20 
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Date 

ONLINE -OFFLlNE_bit /* bit 'on* when auto 
analysis is being done on newly arrived logs */ 
Filepath_List= { f ilepathl , f ilepath2 , 
5 f ilepath3 . . . } /* file path given for restored offline 

logs */ 

Log_Info_Request 

SD_Type 
SD_Name {FQHN} 
10 Date 

Onl ine_Tabl e_Reques t 
O f f 1 ine_Tabl e_Re ques t 

Output to DAM: 
15 Log_Location_Reply 

SD_Type /* type derived from name */ 

SD_Name {FQHN, IP address} 

Date 

Retrieval_Interval 

20 Time 

File_Location_List={f ilepathl, f ilepath2, 

f ilepath3 . . . } 

f ilepath'n' ={ONLINE_bit, ONLINE=f ilepath} 
Log_Info_Reply 
25 SD_Type 

SD_Name {FQHN, IP address} 

LCM_Name 

LC_Name 

Online_Of f line 
3 0 Offline_Date 
Online_Date 
Log_Date 

Retrieval_Interval 
Online_Table_Reply 
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O f f 1 i ne_Tabl e_Rep ly 

Online Log Archival Table 
SD_Type 
5 SD_Name 

IP_address 
LCM_Name 
LC_Name 
Archival_Date 
10 Log_Date 

Retrieval_Interval 

Time= { timel , time2 , time3 . . . } 

Filepath={f ilepathl, f ilepath2, £ilepath3 . . . } 

15 Offline Log Archival Table 

SD_Type 

SD_Name 

IP_address 

LCM_Name 
2 0 LC_Name 

Of f line_Date 

Log_Date 

Retrieval_lnterval 
Time={time 1, time2 , time3} 
25 Filepath={N/A, N/A, N/A} 



Data Analysis Manager (DAM) 

Input from LCM: 
30 Log archival transaction complete 

LCM_Name {FQHN, IP address} 
LC_Name {FQHN, IP address} 
SD_Name {FQHN, IP address} 
Errors 
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LCM_archival_complete /*when all logs have been 
transferred to the SM for that interval*/ 
LCM_status_update 

LC_List={LC_.Namel, LC_Name2 , LC_Name3 . . . } 
5 LC_Name ' n ' = { FQHN , IP address. 

status= [archived | cached | waiting ] } 



Input from WAS: 

Log_Location_Request /* for custom analysis */ 

10 SD_Type 

SD_Name {FQHN, IP address} 
Date„Range={Date | From_To} 
On line= {ONLINE | OFFLINE } 
Of fline_File_Location__List={f ilepathl, 
IS filepath2, f ilepath3 . . . } /.* restored filepath knovm */ 
FUI.L_TEXT = {ON | OFF } 
Custom_Metrics_Request 

Filter_Type= {customized filter keys} 
SD_Type 
SD_Name {FQHN} 
Date_Range={Date | From_To} 
Online_Table_Request 
Of f line_Table_Request 

Input from SM: 

Log_Loc a t i on_Rep ly 
SD_Type 

SD_Name {FQHN. IP address} 
Date 

Retrieval_Interval 
Time 

File_Location_List={f ilepathl , f ilepath2, 
f ilepath3 . . . } 

f ilepath' n' ={ONLINE_bit , ONLlNE= filepath} 



25 



30 



» - 
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Log_Info_Reply 
SDJType 

SD_Name {FQHN, IP address} 
LCM_Name 
LC_Name 

Online_Of f line 
Of f line_Date 
Online_Date 
Log_Date 

Retrieval_Interval 
Onl ine_Tabl e_Reply 
Of f line_Table_Reply • 

Input from DAS: 
15 System_Conf iguration 

Archival_Durat ion= { typel . type2 , type3 . . . } 
type ' n ' = {online= [ number_months ] . 

of f line= [number_months] } 

Retrieval_Interval={default=24 hrs | hourly 

20 interval=l - 24 hrs} 

Cleanup_lnterval={ default=7 days | weekly 

interval=l - 7days) 

SDtypes={ typel, type2, type3 . . . } 
type ' n" = {code, description} 
25 Devicelist={devicel, device2 , device3 . . . } 

Filters={ f iltertypel, filtertype2, 
f iltertype3 . . . } 

filtertype'n'={keyl, key2, key3 . . . } 
Alarms={alarmtypel, alarmtype2 , alarmtype3 . . - } 
3 0 alarmtype'n'^keyl, key2 , key3 . . . } 

LCMlist={lcml , lcm2, lcm3 . . . } 

lcm' n" ={ FQHN, IPaddr, responsibility} 



5 



10 



Output to LCM: 
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SD system configuration file: 

Retrieval_Interval={default=24 hrs | hourly 
interval=l - 24 hrs} 

Cleanup_Interval={ default=7 days | weekly 
5 interval=l - 7days) 

LC_L i s t = { LC_Name 1 , LC_Name 2 , LC_Naroe 3 . • . } 
LC_Name ' n ' ={FQHN, IF address} 

SD_List={SD_Namel, SD_Name 2, SD_Narae 

3. . . ) 

10 SD__Name ' n ' = { FQHN r IP address} 

LCM_status_request /* request status of LC log 
archiving managed by LCM */ 

Output to SM: 
15 System_Conf iguration 

Archival„Duration={typel, type2, type3 . . . } 
type " n ' ={online= [number_raonths] , 
of f line= tnumber_months] } 
Log_Location_Request 
20 SD_Type 

SD_Name {FQHN} 
Date 

ONLINE-OFFLINE_bit /* bit 'on' when auto 
analysis is being done on newly arrived logs */ 
25 Filepath_List={f ilepathl, filepath2, 

f ilepath3 . . . } 

Log_Inf o_Request 
SD_Type 
SD_Name {FQHN} 
30 Date 

Onl ine__Tab 1 e_Reque s t 
Of f line_Table_Request 



Output to WAS: 
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Full_Text_Reply 

Logf ile_Text_Buf f er /* for read-only access */ 
Custom_Metrics_Reply 
Metrics_Table 
5 Status 

Errors 
Alarms 

SearchJResults 
Online_Table_Reply /* summary of logs archived 

10 online */ 

Of f line_Table_Reply /* summary of logs archived 

offline */ 

Output to DAS: 
15 Session_Analysis 

Date= {Month, Day, Year} 

Start_Time 

Session_ID 

Device_Type 
20 Logf ile_Type 

Log f i 1 e_Da t e_T ime 

Retrieval_Interval 
Session_Results 

Da te= {Month, Day, Year} 
25 Completion_Time 

Session_ID 

Device_Type 

Logf ile_Type 

Logf ile_Date_Time 
30 Error_Code 

Alarms={none | [alarml, alarm2 , alarm3 . . . ] } 

Errors={none | [errorl, error2 , error3 . . . ] } 

Metrics={keylresults, key2results , 
key3results. . . } 
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key 'n'results={hitl. hit2 , hit3 . . - } 
Device_Update 

Device_Type 
Devi c e_Name 
5 StatUS= {ACTIVE, HISTORIC} 

Data Analysis Store (DAS) 
Database Schema 

TABLE: analysis„session (used to store information about 
10 the logfile analysis) 



FIELDS: 

session_id /* incorporate the date into the 
sessionid */ 
15 year /* Required for */ 

month /* ease of extraction of */ 
day /* summary metrics.*/ 

device_type (name of firewall contivity switch, 

spam machine ,..'.) 
20 logfile_type (type of file that was parsed, ie. 

some SDs will produce a number of logfiles) 

logfile_date (date and time of logfile) 
retrieval_interval (system log retrieval rate) 
start_time /* required to track DAM-system */ 
25 completion_time /* performance */ 

TABLE: session_alarms 



FIELDS : 

session_id 
30 alarmcode 

status /* status of each alarm - active or 

acknowledged V 

severity 



NOU 27 2002 12:50 FR NORTEL NETWORKS 613 768 3635 TO 6 1 703 305 8568 P. 13/22 

• # 

32 



TABLE: session_errors 



FIELDS : 

session_id 
5 errorcode 

status /* status of each error - active or 
acknowledged */ 

severity 

TABLE: logfile_ types (used to store information about 
10 versions of software e.g., firewall - Raptor 4.0 vs 
Raptor 6.0) 



FIELDS : 

• device_type 
15 logfile_type 



TABLE: metr ic_types (used to store information about the 
metrics that need to be calculated and where to find the 
results) 

20 

FIELDS : 

metric__id (this will be a number from 1-30 
and is the place where the results are stored in the 
tables. For example, if this has a value of 2, then in 
25 the individual results tables the result of this metric 
is stored in the metric2 field. ) 

device_type (ie. 
FIREWALL , SPAM , CONTI VITY , FTPDROPBOX , USER_STATS ) 

logfile_type (e.g. Raptor 4, Raptor 6) 
30 metric_name (this is the name that is used to 

describe the particular metric being found ie . Number of 
FTP connects) 

metric_key (this is the value that is being 
searched ie. ftp. *connection for) 
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status (as we are storing all metrics for many 
years in the database, a particular metric that was used 
in the past may no longer be valid but still requires a 
placeholder in the database for historic data. The 
5 possible entries in this field are ACTIVE, or HISTORIC 
where if the status is ACTIVE, then it will be used for 
analysis) 

TABLE: user_table (used to store information about the 
users accessing this tool) 

10 

FIELDS : 

userid (ie. CN for certs or user id) 

device_type (i.e. 
ALL , FIREWALL » SPAM, CONTIVITY , FTPDROPBOX , USER_STATS ) 
1S type_of_access (e.g. DBA, ANALYST, HELPDESK, 

CORP- INVESTIGATIONS ) 

user_name 

use r_phon e 

TABLE: access (used to store information about the 
20 different levels of access) 



FIELDS : 

type„of_access (e.g. DBA, ANALYST, HELPDESK, 
CORP- INVESTIGATIONS ) 
25 TABLE: special_access (used to determine access rights to 
a log in scenarios where specific, limited access is 
granted) 

FIEKD.S7 - 

30 «ECSerid (ie. CN for certs or userid) 

4evice_name ( i.e. ALL, FQHN ( S ) ) /* required 
for security investigations V 

date (i.e. ALL, DATE RANGE) /* required for 
security investigations */ 
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TABLE: firewall (used to store the metrics gathered on a 
per firewall basis per logfile basid - for the first cut 
there will be one entry per firewall per day but as the 
processing becomes more often, there may be many per 
5 firewall per day. ) 



FIELDS : 

session_id 

metricl to metric 30 (used for counts and sums) 
10 TABLE: f irewall_monthly (used to store firewall 
information but summarized by month) 



FIELDS : 

firewall 
15 year 
month 

metricl to metric 30 
TABLE: f irewall_user (used to store firewall information 
based on the USER_STATS) 

20 

FIELDS: 

t r ansae tion_type - things like connects per 
userid, bytes transferred per userid, etc. This 
information is done on a per firewall per logfile basis) 
25 session_id 

userid 

metricl to metric 30 
TABLE: f irewall_keyword (used to store the matched 
keyword information. This is done on a per firewall per 
30 logfile basis.) 



FIELDS : 

session_id 
search_key 



NOU 2? 2002 12:51 FR NORTEL NETWORKS 613 768 3635 TO 6 1 703 305 8568 P. 16/22 



35 



matched_line (string where the match was found) 
userid (if possible, the userid extracted from 

the matched line) 

count (?) (ongoing count rather than additional 
5 entries in the db?) 

TABLE: contivity (used to store the metrics gathered on a 

per contivity basis per logfile basis) 

FIELDS : 
10 session_id 

metricl to metric 3 0 (counts and sums) 
TABLE: contivity_monthly (used to store contivity 
information but summarized by month) 

15 FIELDS: 

contivity 

year 

month 

metricl to metric 30 
20 TABLE: contivity_user (used to store contivity 
information based on the USER_STATS) 

FIELDS : 

transaction_type (things like connects per 
25 userid, bytes transferred per userid, etc. this 

information is done on a per contivity per logfile basis) 
session_id 
userid 

metricl to metric 30 
30 TABLE: cont ivi ty__keyword (used to store the matched 

keyword information. This is done on a per contivity per 
logfile basis . ) 



FIELDS: 
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session_id 
search_key 

matched_line (string where the match was found) 
userid (if possible, the userid extracted from 
5 the matched line) 

count (?) (ongoing count rather than additional 
entries in the db?) 

TABLE: dropbox (used to store the metrics gathered on a 
per dropbox basis per logfile basis) 

10 

FIELDS : 

session_id 

metricl to metric 30 
TABLE: dropbox_monthly (used to store dropbox information 
IS but summarized by month) 



FIELDS : 

dropbox 
year 

20 month 

metricl to metric 3 0 
TABLE: dropbox^user (used to store firewall information 
based on the USER_STATS) 



25 FIELDS : 

transact ion„ type - things like connects per 
userid, bytes transferred per userid, etc. this 
information is done on a per dropbox per logfile basis) 
session_id 
30 userid 

metricl to metric 3 0 
TABLE: dropbox_keyword (used to store the matched keyword 
information. This is done on a per firewall per logfile 
basis . ) 
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FIELDS : 

session__id 

keyword__key (key that was looked for) 
5 matched_line (string where the match was found) 

userid (if possible, the userid extracted from 
the matched line) 

count (?) (ongoing count rather than additional 
entries in the db?) 
10 TABLE: list_contivity (used to store the list of 
contivities that have information stored in this 
database) 

FIELDS : 

15 device_status (as we are storing metrics for 

many contivities for many years in the database, a 
particular contivity that was used in the past may no 

longer be valid but still requires a 
placeholder in the database for historic data. The 
20 possible entries in this field are ACTIVE , or HISTORIC 
where i f the 

status is ACTIVE, then it will be used for 

analysis) 

device_name 
25 logfile_type 

TABLE: list_dropboxes (used to store the list of 
dropboxes that have information stored in this database) 

FIELDS : 

30 device_status (as we are storing metrics for 

many dropboxes for many years in the database, a 
particular dropbox that was used in the past may no 

longer be valid but still requires a 
placeholder in the database for historic data. The 
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possible entries in this field are ACTIVE, or HISTORIC 
where if the 

status is ACTIVE, then it will be used for 

analysis) 
5 device_name 
logf ile_ type 

TABLE: list_f irewalls (used to store the list of 
firewalls that have information stored in this database) 

FIELDS : 

device_status (as we are storing metrics for 
many firewalls for many years in the database, a 
particular firewall that was used in the past may no 
longer 

be valid but still requires a placeholder in 
the database for historic data. The possible entries in 
this field are ACTIVE, or HISTORIC where if the status is 
ACTIVE, then it will be used for analysis) 
device_name 
logf ile__ type 

TABLE: lis t_keywords (used to store the list of keywords 
that are to be used as part of an analysis) 



15 



FIELDS : 

25 search_key (search string) 

device_type 
logf ile_ type 

responsibility (group who supplied the keyword 
and is responsible to investigate when found - HR (Human 
30 Resources) , NS (Network Security) , CS 
(Corporate Security) ) 

status (as we are storing metrics for many 
firewalls for many years in the database, a particular 
firewall that was used in the past may no longer be valid 
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but still requires a placeholder in the 
database for historic data. The possible entries in this 
field are ACTIVE, or HISTORIC where if the status is 
ACTIVE, then it will be used for analysis) 
5 TABLE: mailshield (used to store mailshield metrics) 



FIELDS : 

session_id 

metricl to metric 30 (sum and counts) 
10 logfile_type 

TABLE: spam_re j ections (used to store top 10 rejection 
types) 

FIELDS : 
15 session_id 

rejectl to rejectlO 
occurrencel to occurrencelO 
TABLE: list_mailshields (used to store the list of 
mailshields that have information stored in this 
20 database) 



FIELDS: 

device_status (as we are storing metrics for 
many mailshields for many years in the database, a 
25 particular mailshield that was used in the past may no 
longer be valid but still requires a 
placeholder in the database for historic data. The 
possible entries in this field are ACTIVE, or HISTORIC 
where if the 

30 status is ACTIVE, then it will be used for 

analysis ) 

de v i c e_name 

TABLE: mai lshield_monthly (used to store mailshield 
» information but summarized by month) 
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FIELDS : 

mailshield 
year 

5 month 

metricl to metric 30 
TABLE: blocked (used to store blocked metrics) 

FIELDS : 
10 session_id 

r ec ipi en t_emai 1 id 

reason (store the reason that the email was 

blocked) 

subject (the subject of the blocked email) 
15 sender 
TABLE : owners 

FIELDS : 

responsibility (ie, HR (Human Resources, NS 
20 (Network Security) . CS (Corporate Security) ) 

contact_name (person to contact when matched) 
userid 

contac t_phone 

contact_email (This is key so that an email can 
25 be sent out, assuming we decide to automate this 
function) 

TABLE: error__list (used to store information about 
possible system errors) 

30 FIELDS: 

errorno 

severity 

description 
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TABLE: alarm_list (used to store information about log 
alarms) 

FIELDS: 
5 alarmcode 
severity 
dsscripti on 

TABLE: device_ types (used to store list of valid 
device_types - these will be hard-coded into this table ) 

10 

FIELDS: 

device_type (i.e. FIREWALL, CONTIVITY, 

SPAM, . . . ) 

TABLE: lcm_list (used to store list of Log Collector 
15 Managers) 

FIELDS : 

de vi c e_name 

responsibility (string - depending on 
20 implementation could be geographic or device type 
dependent ) 

TABLE: sys„config (used to store list of system 
parameters ) 

25 FIELDS: 

retrieval_interval 

c 1 eanup_int erval 

devi c e_t ype 

online_duration 
30 of f line_duration 
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